Display device with a holographic combiner

ABSTRACT

An augmented reality headset may include a reflective holographic combiner to direct light from a light engine into a user&#39;s eye while also transmitting light from the environment. The combiner and engine may be arranged to project light fields with different fields of view and resolution to match the visual acuity of the eye. The combiner may be recorded with a series of point to point holograms; one projection point interacts with multiple holograms to project light onto multiple eye box points. The engine may include a laser diode array, a distribution waveguide, scanning mirrors, and layered waveguides that perform pupil expansion and that emit wide beams of light through foveal projection points and narrower beams of light through peripheral projection points. The light engine may include focusing elements to focus the beams such that, once reflected by the holographic combiner, the light is substantially collimated.

This application is a continuation of International Application No.PCT/US2017/052573, filed Sep. 20, 2017, which claims benefit of priorityto U.S. application Ser. No. 15/709,398, filed Sep. 19, 2017, which isabandon, which claims benefit of priority to U.S. ProvisionalApplication No. 62/397,312, filed Sep. 20, 2016. The above applicationsare incorporated herein by reference. To the extent that any material inthe incorporated application conflicts with material expressly set forthherein, the material expressly set forth herein controls.

BACKGROUND

Virtual reality (VR) allows users to experience and/or interact with animmersive artificial environment, such that the user feels as if theywere physically in that environment. For example, virtual realitysystems may display stereoscopic scenes to users in order to create anillusion of depth, and a computer may adjust the scene content inreal-time to provide the illusion of the user moving within the scene.When the user views images through a virtual reality system, the usermay thus feel as if they are moving within the scenes from afirst-person point of view. Similarly, augmented reality (AR) and mixedreality (MR) combine computer generated information with views of thereal world to augment, or add content to, a user's view of theirenvironment. The simulated environments of VR and/or the enhancedcontent of AR/MR may thus be utilized to provide an interactive userexperience for multiple applications, such as interacting with virtualtraining environments, gaming, remotely controlling drones or othermechanical systems, viewing digital media content, interacting with theinternet, or the like.

However, conventional VR, AR, and MR systems may suffer fromaccommodation-convergence mismatch problems that cause eyestrain,headaches, and/or nausea.

Accommodation-convergence mismatch arises when a VR or AR systemeffectively confuses the brain of a user by generating scene contentthat does not match the depth expected by the brain based on the stereoconvergence of the two eyes of the user. For example, in a stereoscopicsystem the images displayed to the user may trick the eye(s) intofocusing at a far distance while an image is physically being displayedat a closer distance. In other words, the eyes may be attempting tofocus on a different image plane or focal depth compared to the focaldepth of the projected image, thereby leading to eyestrain and/orincreasing mental stress. Accommodation-convergence mismatch problemsare undesirable and may distract users or otherwise detract from theirenjoyment and endurance levels (i.e. tolerance) of virtual reality oraugmented reality environments.

SUMMARY

Various embodiments of an augmented reality (AR), and/or mixed reality(MR) direct retinal projector system that may include an AR headset(e.g., a helmet, goggles, or glasses) that uses a reflective holographiccombiner to direct light from a light engine into the user's eye, whilealso transmitting light from the user's environment to thus provide anaugmented view of reality. The holographic combiner may be recorded witha series of point to point holograms; one projection point interactswith multiple holograms to project light onto multiple eye box points.The holograms may be arranged so that neighboring eye box points areilluminated from different projection points. The holographic combinerand light engine may be arranged to separately project light fields withdifferent fields of view and resolution that optimize performance,system complexity and efficiency, so as to match the visual acuity ofthe eye. The light engine may implement foveal projectors that generallyproject wider diameter beams over a smaller central field of view, andperipheral projectors that generally project smaller diameter beams overa wider field of view.

The light engine may include multiple independent light sources (e.g.,laser diodes, LEDs, etc.) that can independently project from thedifferent projection points, with a proportion being foveal projectorsand a proportion being peripheral projectors. In some embodiments, thelight engine may include two or more two-axis scanning mirrors to scanthe light sources; the light sources are appropriately modulated togenerate the desired image. The light engine may include a series ofoptical waveguides with holographic or diffractive gratings that movethe light from the light sources to generate beams at the appropriateangles and positions to illuminate the scanning mirrors; the light isthen directed into additional optical waveguides with holographic filmlayers recorded with diffraction gratings to expand the projectoraperture and to maneuver the light to the projection positions requiredby the holographic combiner.

In some embodiments, the light engine may include at least one focusingelement (e.g., optical lens, holographic lens, etc.) for each projectorto focus emitted light beams such that, once reflected off theholographic combiner, the light is substantially collimated when itenters the subject's eye. The required focal surface may be complicatedby the astigmatism of the holographic combiner, but is a curved surfacein front of the combiner. The ideal focal surface is different fordifferent eye box positions, and errors may lead to less collimatedoutput. However, in some embodiments, this can be compensated byreducing the beam diameter for different angles where the errors betweenthe ideal focal surface and the actual best fit focal surface aregreatest, which alleviates the problem by increasing the F-number andhence the depth of focus of the beam.

In some embodiments, active beam focusing elements may be provided foreach projection point. This may reduce or eliminate the need to changebeam diameter with angle. This may also enable beams that diverge intothe eye to, rather than being collimated, match the beam divergence ofthe supposed depth of the virtual object(s) being projected by the lightengine.

The AR system may not require extra moving parts or mechanically activeelements to compensate for the eye changing position in the eye box orfor the changing optical power from the holographic combiner during thescan, which simplifies the system architecture when compared to otherdirect retinal projector systems. Further, the holographic combiner maybe implemented by a relatively flat lens when compared to curvedreflective mirrors used in other direct retinal projector systems.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example of different types of eye focus.

FIG. 2 illustrates one embodiment of a conventional near-eye virtualreality system.

FIG. 3 illustrates an example of parallel light beams entering an eye.

FIG. 4 illustrates a direct retinal projector system that uses a curvedellipsoid mirror to direct light from a projector into a subject's eye,while also transmitting light from the environment to the subject's eye.

FIG. 5 illustrates an augmented reality (AR) system that uses areflective holographic combiner to direct light from a light engine intoa subject's eye, while also transmitting light from the environment tothe subject's eye, according to some embodiments.

FIG. 6 illustrates an AR headset that includes a reflective holographiccombiner to direct light from a light engine into a subject's eye, whilealso transmitting light from the environment to the subject's eye,according to some embodiments.

FIG. 7 illustrates high-level components of an AR system, according tosome embodiments.

FIG. 8 illustrates foveal and peripheral projectors of a light engine inan AR headset, according to some embodiments.

FIG. 9 illustrates light beams from foveal projectors in an AR system,according to some embodiments.

FIG. 10 illustrates light beams from peripheral projectors in an ARsystem, according to some embodiments.

FIG. 11 illustrates foveal and peripheral eye boxes for an AR system,according to some embodiments.

FIGS. 12A through 12C illustrate a laser array for an AR system,according to some embodiments.

FIGS. 13A through 13C illustrate collimating lenses for a laser array inan AR system, according to some embodiments.

FIG. 14 illustrates a laser array projector, according to someembodiments.

FIG. 15 illustrates a laser array projector and waveguide withholograms, according to some embodiments.

FIG. 16 illustrates a 2D scanning microelectromechanical systems (MEMS)mirror, according to some embodiments.

FIG. 17 illustrates foveal waveguides, according to some embodiments.

FIG. 18 illustrates beam angles in cosine space for foveal waveguides,according to some embodiments.

FIG. 19 illustrates peripheral waveguides, according to someembodiments.

FIG. 20 illustrates beam angles in cosine space for peripheralwaveguides, according to some embodiments.

FIG. 21 further illustrates peripheral waveguides, according to someembodiments.

FIGS. 22A through 22C are graphs illustrating angular selectivity for aholographic combiner, according to some embodiments.

FIG. 23 illustrates foveal projections for a holographic combiner,according to some embodiments.

FIG. 24 illustrates peripheral projections for a holographic combiner,according to some embodiments.

FIG. 25 illustrates a best fit focus curve and a focusing element forperipheral projections in an AR system, according to some embodiments.

FIG. 26 is a graph of peripheral projector resolution vs. pupil angle inan AR system, according to some embodiments.

FIG. 27 illustrates a best fit focus curve and a focusing element forfoveal projections in an AR system, according to some embodiments.

FIG. 28 illustrates projector scan angle for foveal projections,according to some embodiments.

FIG. 29A is a graph of foveal projector resolution vs. pupil angle in anAR system, according to some embodiments.

FIG. 29B is a graph of beam diameter for foveal projections in an ARsystem, according to some embodiments.

FIG. 30 is a high-level flowchart of a method of operation for an ARsystem as illustrated in FIGS. 5 through 29B, according to someembodiments.

This specification includes references to “one embodiment” or “anembodiment.” The appearances of the phrases “in one embodiment” or “inan embodiment” do not necessarily refer to the same embodiment.Particular features, structures, or characteristics may be combined inany suitable manner consistent with this disclosure.

“Comprising.” This term is open-ended. As used in the claims, this termdoes not foreclose additional structure or steps. Consider a claim thatrecites: “An apparatus comprising one or more processor units . . . .”Such a claim does not foreclose the apparatus from including additionalcomponents (e.g., a network interface unit, graphics circuitry, etc.).

“Configured To.” Various units, circuits, or other components may bedescribed or claimed as “configured to” perform a task or tasks. In suchcontexts, “configured to” is used to connote structure by indicatingthat the units/circuits/components include structure (e.g., circuitry)that performs those task or tasks during operation. As such, theunit/circuit/component can be said to be configured to perform the taskeven when the specified unit/circuit/component is not currentlyoperational (e.g., is not on). The units/circuits/components used withthe “configured to” language include hardware—for example, circuits,memory storing program instructions executable to implement theoperation, etc. Reciting that a unit/circuit/component is “configuredto” perform one or more tasks is expressly intended not to invoke 35U.S.C. § 112, paragraph (f), for that unit/circuit/component.Additionally, “configured to” can include generic structure (e.g.,generic circuitry) that is manipulated by software or firmware (e.g., anFPGA or a general-purpose processor executing software) to operate inmanner that is capable of performing the task(s) at issue. “Configureto” may also include adapting a manufacturing process (e.g., asemiconductor fabrication facility) to fabricate devices (e.g.,integrated circuits) that are adapted to implement or perform one ormore tasks.

“First,” “Second,” etc. As used herein, these terms are used as labelsfor nouns that they precede, and do not imply any type of ordering(e.g., spatial, temporal, logical, etc.). For example, a buffer circuitmay be described herein as performing write operations for “first” and“second” values. The terms “first” and “second” do not necessarily implythat the first value must be written before the second value.

“Based On” or “Dependent On.” As used herein, these terms are used todescribe one or more factors that affect a determination. These terms donot foreclose additional factors that may affect a determination. Thatis, a determination may be solely based on those factors or based, atleast in part, on those factors. Consider the phrase “determine A basedon B.” While in this case, B is a factor that affects the determinationof A, such a phrase does not foreclose the determination of A from alsobeing based on C. In other instances, A may be determined based solelyon B.

“Or.” When used in the claims, the term “or” is used as an inclusive orand not as an exclusive or. For example, the phrase “at least one of x,y, or z” means any one of x, y, and z, as well as any combinationthereof.

DETAILED DESCRIPTION

Various embodiments of an augmented reality (AR), and/or mixed reality(MR) direct retinal projector system are described that may, forexample, resolve the convergence-accommodation conflict in head-mountedAR, MR, and VR systems. While at least some embodiments may providemixed reality, for simplicity the system may generally be referred toherein as an AR system. Embodiments of an AR headset (e.g., a helmet,goggles, or glasses) are described that may include or implementdifferent techniques and components of the AR system. In someembodiments, an AR headset may include a reflective holographic combinerto direct light from a projector light engine into the user's eye, whilealso transmitting light from the user's environment to thus provide anaugmented view of reality. In some embodiments, the holographic combinermay be recorded with a series of point to point holograms; oneprojection point interacts with multiple holograms to project light ontomultiple eye box points. In some embodiments, the holograms are arrangedso that neighboring eye box points are illuminated from differentprojection points.

In some embodiments, the holographic combiner and light engine may bearranged to separately project light fields with different fields ofview and resolution that optimize performance, system complexity andefficiency, so as to match the visual acuity of the eye. In someembodiments, the light engine may include foveal projectors thatgenerally project wider diameter beams over a smaller central field ofview, and peripheral projectors that generally project smaller diameterbeams over a wider field of view.

In some embodiments, the light engine may include multiple independentlight sources (e.g., laser diodes, LEDs, etc.) that can independentlyproject from the different projection points, with a proportion beingfoveal projectors and a proportion being peripheral projectors. In someembodiments, the light engine includes two or more two-axis scanningmirrors to scan the light sources; the light sources are appropriatelymodulated to generate the desired image. In some embodiments, the lightengine includes a series of optical waveguides with holographic ordiffractive gratings that move the light from the light sources togenerate beams at the appropriate angles and positions to illuminate thescanning mirrors; the light is then directed into additional opticalwaveguides with holographic film layers recorded with diffractiongratings to expand the projector aperture and to maneuver the light tothe projection positions required by the holographic combiner.

In some embodiments, the light engine includes a lens for each projectorto focus emitted light beams such that, once reflected off theholographic combiner, the light is substantially collimated again whenit enters the subject's eye. The required focal surface may becomplicated by the astigmatism of the holographic combiner, but is acurved surface in front of the combiner. The ideal focal surface isdifferent for different eye box positions, and errors may lead to lesscollimated output. However, in some embodiments, this can be compensatedby reducing the beam diameter for different angles where the errorsbetween the ideal focal surface and the actual best fit focal surfaceare greatest, which alleviates the problem by increasing the F-numberand hence the depth of focus of the beam. In some embodiments, thesefeatures may be incorporated into a holographic lens.

In some embodiments, active beam focusing elements may be provided foreach projection point. This may reduce or eliminate the need to changebeam diameter with angle. This may also enable beams that diverge intothe eye to, rather than being collimated, match the beam divergence ofthe supposed depth of the virtual object(s) being projected by the lightengine.

With the methods and apparatus presented above, the AR system may notrequire extra moving parts or mechanically active elements to compensatefor the eye changing position in the eye box or for the changing opticalpower from the holographic combiner during the scan, which simplifiesthe system architecture when compared to other direct retinal projectorsystems.

Accommodation and Convergence in AR/VR Systems

The human brain typically uses two cues to gauge distance: accommodation(i.e., eye focus) and eye convergence (i.e., the stereoscopicperspective difference between the two eyes). Conventional near-eyesystems typically use separate miniature screens for each respective eyeto project the images intended for the left eye and the right eye, aswell as optics to allow a user to comfortably focus the eyes at a fardistance during viewing of the left eye and right eye images.Conventional near-eye systems thus produce conflicting visual cues sincethe resulting three-dimensional (3D) image produced by the braineffectively appears at a convergence distance that is closer than theaccommodation distance that each eye focuses on separately, therebyleading to the possibility of headache and/or nausea over time. Heavyusers of conventional systems may potentially train themselves tocompensate for accommodation-convergence mismatch, but a majority ofusers might not.

AR systems typically add information and graphics to an existing scenebeing viewed by a user. In some embodiments, AR may be a powerfulexperience, since the user can see both the projected images and/orsprites (i.e., the augmented world) as well as the surrounding scene(i.e., the real world) directly through the AR system rather than usingcamera systems to project a version of the surrounding scene lessaccurately onto screen displays for each eye.

FIG. 1 depicts an example of different types of eye focus. In system 100of FIG. 1, an eye 110A may be selectively configured to focus at a fardistance, as shown by the incident light originating from a distantlocation and focusing onto the retina (i.e., the back internal surface)of eye 110A by the internal lens of eye 110A. In another embodiment, eye110A may instead be selectively configured for a close focus scenario,as shown by light from a nearby location being incident upon the eye andfocusing onto the retina.

FIG. 2 illustrates one embodiment of a conventional near-eye system 200.As depicted, right eye 210 and left eye 220 are focused on a focal plane230 where an image for right eye 240 and an image for left eye 250,respectively, are displayed. As right eye 210 and left eye 220 focus ontheir respective images on focal plane 230, the brain of the usercombines the images into a resulting 3D image 260. In one embodiment,the accommodation distance may be the distance between focal plane 230and an eye of the user (e.g., right eye 210 and/or left eye 220), andthe convergence distance may be the distance between resulting 3D image260 and an eye of the user. Since, as depicted in FIG. 2, theaccommodation distance differs from the convergence distance,conventional near-eye system 200 therefore results in anaccommodation-convergence mismatch and may cause discomfort for the useras described above.

FIG. 3 illustrates an example of parallel light beams entering an eye300. As shown, various sets of parallel light beams that enter eye 300are focused by eye 300 such that the parallel beams within a respectiveset land at the same place on the retina of eye 300.

Direct Retinal Projector System with Scanning Mirror and EllipsoidMirror

A direct retinal projector system may be implemented as a headset (e.g.,a helmet, goggles, or glasses) that includes a scanning projector,curved ellipsoid mirror, gaze tracking technology, and a secondaryscanning mirror. FIG. 4 illustrates a direct retinal projector system400 that scans virtual reality (VR) images, pixel, by pixel, to asubject's eye 490. In some embodiments, of a direct retinal projectorsystem 400, under control of a controller (not shown), light beams arescanned by a scanning projector 402 to a secondary scanning mirror 404,and the light beams are then scanned by the scanning mirror 404 todifferent positions on a curved ellipsoid mirror 406 in front of thesubject's eye 490 according to the current position of the subject's eye490 as determined by gaze tracking technology (not shown), and reflectedoff the curved mirror 406 through the subject's pupil 492 to form theimages on the subject's retina to thus provide a VR view to the subject.Unlike conventional screen-based VR/AR systems, there is no intermediateimage on a screen or surface that the subject views. The direct retinalprojector system 400 may at least partially eliminate eye lensaccommodation from the retinal projection focus to help eliminate theaccommodation convergence mismatch. In some embodiments, to provide anAR or MR experience to the user, the curved mirror 406 may allow lightfrom the subject's environment to pass through the mirror to thesubject's eye 490 while simultaneously reflecting the light beamsgenerated by the projector 402 to the subject's eye 490, thus enablingthe subject to see elements of both an external (real) scene and thevirtual reality (VR) images projected by the projector. Note that thedirect retinal projector system 400 is shown for only one eye; generallybut not necessarily, there will be a second direct retinal projectorsystem 400 for the second eye.

In the direct retinal projector system 400 as illustrated in FIG. 4, thecurved ellipsoid mirror 406 bulges outward significantly, and thereforethe headset may be cumbersome and odd looking when worn by a user. Inaddition, the projector 402 may emit relatively small beams (e.g., 1 mmdiameter) that may limit resolution, and the system 400 may have arelatively limited field of view. In addition, the system ismechanically complex; for example, the secondary scanning mirror 404 foradjusting for different eye positions adds complexity. Further, theprojector 402 and scanning mirror 404 may be relatively large, furtheradding to the bulk of the headset.

Direct Retinal Projector AR System with Holographic Combiner

Embodiments of a direct retinal projector AR system are described thatinclude an AR headset with reflective holographic combiners to directlight from light engines into the user's eyes, while also transmittinglight from the user's environment to thus provide an augmented or mixedview of reality. The holographic combiners may be implemented asholographic films on relatively flat lenses when compared to the curvedellipsoid mirrors 406 of the system 400 as illustrated in FIG. 4, andthus do not bulge as do the mirrors 406 in that system, making theheadset less bulky, more comfortable to wear, and more normal looking;the headset may, for example, be implemented as a relativelynormal-looking pair of glasses. Further, embodiments of the AR systemmay not require extra moving parts or mechanically active elements suchas scanning mirror 404 to compensate for the eye changing position inthe eye box or for changing optical power from the holographic combinerduring the scan, which greatly simplifies the system architecture whencompared to the direct retinal projector system of FIG. 4. Further, thelight engine may include hologram-based foveal projectors that generallyproject wider diameter beams over a smaller central field of view, andhologram-based peripheral projectors that generally project smallerdiameter beams over a wider field of view. Thus, the light engine may beimplemented as a relatively small and thin solid-state system, furtherreducing the mechanical complexity and bulk of the AR headset whencompared to a system as illustrated in FIG. 4.

FIGS. 5 through 30 illustrate architecture, components, and operation ofexample embodiments of a direct retinal projector AR system. FIG. 30 isa high-level flowchart of a method of operation for an AR system asillustrated in FIGS. 5 through 29B, according to some embodiments.Elements of FIG. 30 are explained in greater detail in FIGS. 5 through29. As indicated at 3010, a laser array projector emits light beams toentrance points of a distribution waveguide under control of acontroller. As indicated at 3020, a distribution waveguide guides thelight to respective exit points; the light is emitted to scanningmirrors. In some embodiments, the entrance and exit points may beimplemented as holograms using a holographic film. Alternatively, theentrance and exit points may be implemented as surface relief gratings(SRG), which are typically created using lithographic techniques ratherthan a holographic film. As indicated at 3030, the scanning mirrors scanthe light to layered waveguides with pupil expansion. As indicated at3040, the layered waveguides expand the light and project the expandedlight beams from respective projection points. As indicated at 3050,focusing elements focus the projected beams in front of a holographiccombiner. As indicated at 3060, point-to-point holograms of theholographic combiner redirect the light beams to respective eye boxpoints. In some embodiments, the subject's pupil position may betracked, and the AR system may selectively illuminate different eye boxpoints according to the tracking information.

FIGS. 5 and 6 illustrate an augmented reality (AR) system 500 that usesa reflective holographic combiner 550 to direct light projected from alight engine 510 into a subject's eye 590, while also transmitting lightfrom the environment to the subject's eye 590, according to someembodiments. In some embodiments, the AR system 500 may include aheadset (e.g., a helmet, goggles, or glasses as shown in FIG. 6) thatincludes a frame (not shown in FIG. 5), a light engine 510, a gazetracking 504 component, and a lens that includes a holographic combiner550, for example implemented as a holographic film on either side of, orembedded in, the lens. The lens may be a piece of curved glass orplastic with optical power depending on the user's particularrequirements, or alternatively a piece of curved glass or plastic withno optical power. Note that, for simplicity, the system 500 is shown foronly one eye; generally but not necessarily, there will be a secondsystem 500 (light engine 510, gaze tracking 504, and lens withholographic combiner 550) for the second eye. However, there may be asingle controller 502 in the system 500.

The system 500 may include a controller 502 that controls operation ofthe light engine(s) 510. The controller 502 may be integrated in theheadset, or alternatively may be implemented at least in part by adevice (e.g., a personal computer, laptop or notebook computer,smartphone, pad or tablet device, game controller, etc.) external to theheadset and coupled to the headset via a wired or wireless (e.g.,Bluetooth) connection. The controller 502 may include one or more ofvarious types of processors, CPUs, image signal processors (ISPs),graphics processing units (GPUs), coder/decoders (codecs), memory,and/or other components. The controller 502 may, for example, generatevirtual content for projection by the light engines 510 of the headset.The controller 502 may also direct operation of the light engines 510,in some embodiments based at least in part on input from a gaze tracking504 components of the headset. The gaze tracking 504 component may beimplemented according to any of a variety of gaze tracking technologies,and may provide gaze tracking input to the controller 500 so thatprojection by the light engine 510 can be adjusted according to currentposition of the subject's eyes 590.

In some embodiments, the holographic combiner 550 may be recorded with aseries of point to point holograms; one projection point interacts withmultiple holograms to project light onto multiple eye box 560 points. Insome embodiments, the holograms are arranged so that neighboring eye box560 points are illuminated from different projection points. In someembodiments, the holographic combiner 550 and light engine 510 may bearranged to separately project light fields with different fields ofview and resolution that optimize performance, system complexity andefficiency, so as to match the visual acuity of the eye. In someembodiments, the light engine 510 may include one or more (e.g., four,in some embodiments) foveal projectors that generally project widerdiameter beams over a smaller central field of view, and one or more(e.g., four, in some embodiments) peripheral projectors that generallyproject smaller diameter beams over a wider field of view.

In some embodiments, the light engine 510 may include multipleindependent light sources (e.g., laser diodes, LEDs, etc.) that canindependently project from the different projection points (e.g., eightprojection points), with a proportion (e.g., four) being fovealprojectors and a proportion (e.g., four) being peripheral projectors. Insome embodiments, the light engine 510 may include two or more two-axisscanning mirrors to scan the light sources; the light sources may beappropriately modulated (e.g., by controller 502) to generate thedesired image. In some embodiments, the light engine 510 includes aseries of optical waveguides with holographic or diffractive gratingsthat move the light from the light sources to generate beams at theappropriate angles and positions to illuminate the scanning mirrors; thelight is then directed into additional optical waveguides withholographic film layers recorded with diffraction gratings to expand theprojector aperture and to maneuver the light to the projection positionsrequired by the holographic combiner 550.

In some embodiments, the light engine 510 includes a lens for eachprojector to focus collimated light such that, once reflected off theholographic combiner 550, the light is substantially collimated againwhen it enters the subject's eye 590. The required focal surface may becomplicated by the astigmatism of the holographic combiner 550, but is acurved surface in front of the combiner 550. The ideal focal surface isdifferent for different eye box 560 positions, and errors may lead toless collimated output. However, in some embodiments, this can becompensated by reducing the beam diameter for different angles where theerrors between the ideal focal surface and the actual best fit focalsurface are greatest, which alleviates the problem by increasing theF-number and hence the depth of focus of the beam. In some embodiments,these features may be incorporated into a holographic lens for eachprojector.

In some embodiments, active beam focusing elements may be provided foreach projection point. This may reduce or eliminate the need to changebeam diameter with angle. This may also enable beams that diverge intothe eye 590 to, rather than being collimated, match the beam divergenceof the supposed depth of the virtual object(s) being projected by thelight engine 510.

With the methods and components described above, the AR system 500 maynot require extra moving parts or mechanically active elements tocompensate for the eye changing position in the eye box or for thechanging optical power from the holographic combiner during the scan,which greatly simplifies the system architecture when compared to otherdirect retinal projector systems. In addition, embodiments of the ARsystem 500 may provide a wide field of view (FOV) (e.g., ˜120 deg×80 degat the center of the eye box 560), high resolution (e.g., 60 PPD at thefovea of the eye 590). In addition, the holographic combiner 550implemented similar to a regular glasses lens and the very thin lightengine 510 (e.g., <4 mm thick, as shown in FIG. 6) allows the headset tobe implemented in a small package size.

Table 1 lists some example values for parameters of an example AR system500 as illustrated in FIGS. 5 and 6, and is not intended to be limiting.

TABLE 1 Parameter Foveal Peripheral Notes Laser beam diameter 2.3 mm 0.5mm Beam diameter varies at pupil max max with eye direction Resolution60 ppd 13 ppd Resolution reduces gradually for eye angles greater than~3 deg Frame rate 30 Hz 110 Hz Two identical MEMS mirrors with frequency22 kHz, driven differently FOV at eye (one 10° × 10° 120° × 74° At thecenter of the pupil position) eye box (peripheral FOV reduces towardsthe extremes) Range of pupil +/−15° any Eye box size positions direction18.4 mm × 11.5 mm, hence no mechanism needed for inter- pupillarydistance Depth of focus for 3.5 m to infinity user (beam focus adjustedto object distance

Referring to FIGS. 5 and 6, the following provides further non-limitingdetails about and descriptions of the components of the AR system 500.

-   -   The holographic combiner 550 may reduce or eliminate the bulge        of the ellipsoid mirror of a system 400 as illustrated in        FIG. 4. While holographic combiners are used, as an alternative        optical waveguide combiners could be used. However, a reflective        holographic combiner has a better FOV than a waveguide combiner,        which would be limited by total internal reflection angles.        Field of view (FOV) of the holographic combiner 550 may, for        example, be 120 degrees. A curved lens with a holographic film        that implements the holographs may be used as the holographic        combiner 550.    -   By using optical waveguides, the light engine 510 may be very        thin for example ˜3.8 mm.    -   Resolution of the foveal projector (+/−10 deg), 60 PPD around        center. The beam diameter reduces from 3-10 deg. ˜20 PPD at 10        deg.    -   Resolution of the peripheral projector (approximately +60-30        deg), 13 PPD at 10 deg, dropping to a minimum of 2.3 PPD at 50        deg.    -   Holographic combiner        -   Eye box: 18.4×11.5 mm.        -   Point-to-point holographic combiner.        -   8 projection points (4 foveal, 4 peripheral).        -   40 foveal points in the eye box.        -   84 peripheral points in the eye box.        -   Hence, a total of 372 holograms (RGB).    -   Laser Projectors        -   20 laser diodes of each color (RGB)—total 60.        -   16 foveal projectors (laser diodes)—four for each projection            point.        -   4 peripheral projectors (laser diodes)—one for each            projection point.        -   Packaged in a 1D array of edge emitter laser diodes.            However, VCSELs or other light-emitting technologies may be            used in some embodiments.    -   Gaze tracking—In some embodiments, to reduce complexity, gaze        tracking may not be included in the system 500. However, some        embodiments may include a gaze tracking 504 component.

The architecture, components, and operation of an example AR system 500as broadly illustrated in and described for FIGS. 5 and 6 are discussedbelow in greater detail in reference to FIGS. 6 through 30.

FIG. 7 illustrates high-level components of an AR system 500, accordingto some embodiments. FIG. 7 also graphically illustrates the path oflight through the components of an AR system 500, according to someembodiments. As shown in FIG. 7, light emitted by a laser arrayprojector 700 is guided to 2D scanning microelectromechanical systems(MEMS) mirrors 720 by a distribution optical waveguide 710. The lightenters the distribution waveguide 710 at entrance points and exits thewaveguide 710 at exit points. In some embodiments, the entrance and exitpoints may be implemented as holograms using a holographic film.Alternatively, the entrance and exit points may be implemented assurface relief gratings (SRG), which are typically created usinglithographic techniques rather than a holographic film. The mirrors 720scan the light to layered waveguides 730 that perform pupil expansion;the light is then projected from the layered waveguides 730 by fovealand peripheral holographic projectors. In some embodiments, aholographic lens and aperture (one for each projector) focuses theemitted light beams at a focus curve in front of the holographiccombiner 750; the holograms of the reflective holographic combiner 750direct the light beams to respective foveal and peripheral positions onthe eye box to thus scan the light beams to the user's eye 790.

FIG. 8 illustrates foveal and peripheral projectors of an example lightengine in an example AR headset 800, according to some embodiments. Insome embodiments, an AR headset 800 may include a frame 502 (e.g., aneyeglasses frame), a light engine 510, and a holographic combiner 550.FIG. 8 shows a light engine 510 and a reflective holographic combiner550 for only the right eye; however, generally there will also be alight engine 510 and a holographic combiner 550 for the left eye. Insome embodiments, AR headset 800 may include other components, such as agaze tracking component and a controller. In some embodiments of a lightengine 510, there may be four foveal projectors 512 and four peripheralprojectors 514 that each scans beams to the holographic combiner 550;the holographic combiner 500 directs the beams from the projectors 512and 514 to respective foveal positions (the larger squares) andperipheral positions (the smaller squares) of the eye box 560. FIGS. 9and 10 describe foveal 512 and peripheral 514 projectors in more detail.

FIG. 9 illustrates light beams from foveal projectors in an example ARsystem, according to some embodiments. In some embodiments, there may befour foveal projectors 512. For simplicity, FIG. 9 shows views of one ofthe four foveal projectors 512 scanning to one eye box 560 fovealposition. Note, however, that each foveal projector 512 maysimultaneously scan to two or more, or all, of its respective eye box560 positions, and that all of the foveal projectors 512 maysimultaneously scan to their respective eye box 560 positions. However,the actual image that is projected to the eye box 560 may depend on theposition of the subject's eye as determined by a gaze tracking componentof the system 500 (not shown in FIG. 9). In the example shown in FIG. 9,foveal projectors 2 and 3 each scan to 12 respective eye box 560positions, while foveal projectors 1 and 4 each scan to 8 respective eyebox 560 positions.

In some embodiments, each foveal projector 512 may project light beams900 that are 4 mm or greater in diameter. In some embodiments, foveallight beams of 7 mm or greater diameter may be used to generateapproximately 2.3 mm beams entering the subject's pupil. Note that 2.3mm roughly corresponds to the diameter of the pupil at high resolutionunder normal/bright lighting conditions; thus, the foveal light beams900 may substantially fill the subject's pupil to achieve maximumresolution. A large diameter (4 mm or greater, for example 7 mm in someembodiments) for the foveal light beams 900 may thus be used to maximizeresolution at the fovea. The holographic combiner 550 has optical power,and the large diameter of the light beams 900 from the foveal projectors512 may be necessary due to the angle between the light engine 510 andthe combiner 550 and the optical power of the combiner 550 to generate˜2.3 mm, substantially collimated beams directed to the eye box 560 bythe combiner 550. Note, however, that in some embodiments foveal lightbeams within the range of 4 mm-7 mm may be used to achieve adequate, butnot optimal, resolution. As shown in FIG. 9, the foveal beams 900 may berectangular or square. Using rectangular or square beams 900 from thefoveal projectors 512 may help to form a tessellated pattern at the eyebox 560. However, other beam shapes (e.g., circular) may be used in someembodiments.

In some embodiments, each foveal projector 512 may be illuminated by 12laser diodes of the light engine 510; the 12 laser diodes include 4laser diodes of each color/wavelength (red, green, and blue (RGB)).Using 4 laser diodes for each color to illuminate a foveal projector 512may reduce the required scan angle of the scanning MEMS mirrors, andhence reduce the mirror speed required when scanning pixels. Highermirror speeds may tend to smear pixels, which reduces resolution.

In some embodiments, the light engine 510 includes a holographic lensand aperture (one for each foveal projector 512) that focuses theemitted light beams 900 at a focus curve 902 in front of the holographiccombiner 550; the holograms of the reflective holographic combiner 550direct the light beams to respective foveal positions on the eye box 560to thus scan the light beams to the user's eye.

FIG. 10 illustrates light beams from peripheral projectors in an exampleAR system, according to some embodiments. In some embodiments, there maybe four peripheral projectors 514. FIG. 10 shows views of each of thefour peripheral projectors 514 scanning to three eye box 560 peripheralpositions (the smaller squares). Note that each peripheral projector 514may simultaneously scan to two or more, or all, of its respective eyebox 560 positions, and that all of the peripheral projectors 514 maysimultaneously scan to their respective eye box 560 positions. However,the actual image that is projected to the eye box 560 may depend on theposition of the subject's eye as determined by a gaze tracking componentof the system 500 (not shown in FIG. 10). In the example shown in FIG.10, peripheral projectors 1 and 2 each scan to 24 respective eye box 560positions, while peripheral projectors 3 and 4 each scan to 18respective eye box 560 positions. Note that the peripheral eye boxpositions lie within or overlap the foveal eye box positions/squares inthe eyebox 560.

In some embodiments, the projection area on the combiner 550 is fixed.Hence, the projected field of view (FOV) does change depending on theeye position in the eye box 560. In some embodiments, at near the centerof the eye box 560, FOV for the right eye may be:

Azimuth: +60 deg (temporal) to −30 deg (nasal)

Elevation: +35 deg (top) to −38 deg (bottom)

FIG. 11 illustrates foveal 1162 and peripheral 1164 eye boxes for anexample AR system, according to some embodiments. In some embodiments,for the foveal eye box 1162, beam width may be 2.3 mm. Given a beamwidth of 2.3 mm, if the eye's pupil diameter>4.6 mm, then in a worstcase scenario there could be a conflict of information from neighboringfoveal eye box points or positions corresponding to a given fovealprojector. In some embodiments, for the peripheral eye box 1164, beamwidth may be 0.6 mm, with a 1.6 mm pitch between peripheral positions.Given a beam width of 0.6 mm and pitch of 1.6 mm, if the eye's pupildiameter>4.0 mm, then in a worst case scenario there could be a conflictof information from neighboring peripheral eye box points or positionscorresponding to a given peripheral projector. However, note that pupildiameter is typically <4 mm under normal to bright lighting conditions.

FIGS. 12A through 12C illustrate a laser array that may be used in alight engine of an example AR system, according to some embodiments.FIG. 12A shows a 1D array 1200 of laser diodes 1202, e.g. edge emittinglasers. In some embodiments, the laser diodes 1202 may be very lowerpower diodes. FIG. 12B shows the laser diodes 1202 in array 1200emitting full width at half maximum (FWHM) beams 1204. FIG. 12C showsthe laser diodes 1202 in array 1200 emitting max envelope beams 1206.The laser diodes 1202 in an array 1200 may all emit light in the samebandwidth/color, e.g. red, green, or blue light. In some embodiments,there may be 20 laser diodes 1202 in a given array 1202, with one laserdiode 1202 for each peripheral projector of the light engine, and fourlaser diodes 1202 for each foveal projector of the light engine.However, more or fewer diodes 1202 may be used in some embodiments. Forexample, in some embodiments, two laser diodes may be used for eachfoveal projector, and therefore there may be only 12 laser diodes 1202in an array 1200.

In some embodiments, the laser cavities may be rectangular, and thus thebeams emitted by the laser diodes 1202 may not be circular. In someembodiments, the beams emitted by the laser diodes 1202 in an array maybe collimated in two stages.

FIGS. 13A through 13C illustrate collimating lenses for a laser array inan AR system, according to some embodiments. In some embodiments,substantially cylindrical collimating lenses 1304, for example formed ofmolded plastic or glass material, may be used with a laser array 1300;the collimating lenses 1304 may act to collimate light emitted by thediodes 1302 so that the light forms a collimated beam. In someembodiments, holographic elements may be used to collimate the lightemitted by diodes 1302 instead of the molded lenses 1304 shown in FIGS.13A through 13C.

FIG. 14 illustrates a laser array projector 1400 for a light engine inan AR system, according to some embodiments. In some embodiments, alaser array projector 1400 may include laser arrays 1402 and photodiodearrays 1420 on a ceramic substrate 1410. In some embodiments, the laserarray projector 1400 may include three laser arrays 1402, for examplelaser arrays as illustrated in FIGS. 12A through 12C, with one laserarray for each color (red, green, and blue (RGB). In some embodiments,the laser array projector 1400 may include collimating lenses 1404 foreach laser array 1402 as illustrated in FIGS. 13A through 13C. In someembodiments, the laser array projector may include photodiode arrays1420 for each laser array 1402 that monitor light intensity ofrespective laser diodes in the laser arrays 1402. Light intensity varieswith temperature, and so the photodiodes may be used in a feedback loopto maintain a threshold level of intensity for light output by the laserdiodes in the arrays 1402.

In some embodiments, assuming a light engine with four peripheral andfour foveal projectors as shown in FIG. 8, each laser array 1402includes twenty laser diodes that all emit the same color of light (red,green, or blue). In each laser array 1402, there is one laser diode ofthe respective color for each of the four peripheral projectors, andfour laser diodes of the respective color for each of the four fovealprojectors. Thus, there are three laser arrays 1402 in a laser arrayprojector 1410, with one including red-emitting lasers, one includinggreen-emitting lasers, and one including blue-emitting lasers. There are60 laser diodes total in the laser array projector, with twelve (four ofeach color) laser diodes for each of the four foveal projectors (48total), and three (one of each color) laser diodes for each of the fourperipheral projectors (12 total).

While not shown in FIG. 14, a controller of the AV system mayselectively activate and modulate the laser diodes in the laser arrayprojector 1410 to generate light for each color of each RGB pixel thatis being scanned by the system to the subject's respective eye.

FIG. 15 illustrates a laser array projector and distribution opticalwaveguide with holograms that may be used in an example light engine,according to some embodiments. As shown in FIG. 15, a laser arrayprojector 1400 as illustrated in FIG. 14 may be attached or mounted to adistribution waveguide 1500. The distribution waveguide 1500 may beplanar optical waveguide, or alternatively may be an etched opticalwaveguide. Assuming 60 laser diodes (20 of each color) in projector1400, there are 60 entrance points and 60 exit points on waveguide 1500.Light from the laser array projector 1400 enters the distributionwaveguide 1500 at entrance point and exits the waveguide 1500 atcorresponding exit points. In some embodiments, the entrance and exitpoints may be implemented as holograms using a holographic film.Alternatively, the entrance and exit points may be implemented assurface relief gratings (SRG), which are typically created usinglithographic techniques rather than a holographic film. The laser diodesin the projector 1400 each line up with one of the entrance holograms ofthe distribution waveguide 1500, and each laser diode is configured toproject light into its corresponding entrance hologram. The waveguide1500 is configured to guide the light emitted by the laser diodes fromtheir respective entrance holograms to respective exit holograms.

In some embodiments, assuming a light engine with four peripheral andfour foveal projectors as shown in FIG. 8, there are twelve (four ofeach color) entrance holograms on distribution waveguide 1500 for eachof the four foveal projectors (48 total), and three (one of each color)entrance holograms on distribution waveguide 1500 for each of the fourperipheral projectors (12 total). Likewise, there are twelve (four ofeach color) exit holograms on distribution waveguide 1500 for each ofthe four foveal projectors (48 total), and three (one of each color)exit holograms on distribution waveguide 1500 for each of the fourperipheral projectors (12 total). As shown in FIG. 15, in someembodiments, there may be four peripheral exit points corresponding tothe peripheral projectors, each peripheral exit point including one red,one green, and one blue peripheral exit hologram for the respectiveperipheral projector, and sixteen foveal exit points, each foveal exitpoint including one red, one green, and one blue foveal exit hologram.The foveal exit points may be arranged in clusters of four; thus thereare four clusters corresponding to the four foveal projectors. Note thatexit holograms for red, green, and blue light may be overlaid or stackedat each foveal and peripheral exit point.

Light exiting the peripheral exit points and clusters of foveal exitpoints of the distribution waveguide 1500 enters 2D scanningmicroelectromechanical systems (MEMS) mirrors (also referred to asscanning mirrors). In some embodiments, there may be two scanningmirrors for foveal projection, with a first scanning mirror for two ofthe foveal projectors, and a second scanning mirror for the other twofoveal projectors. In some embodiments, there may be two scanningmirrors for peripheral projection, with a first scanning mirror for twoof the peripheral projectors, and a second scanning mirror for the othertwo peripheral projectors.

FIG. 16 illustrates a 2D scanning microelectromechanical systems (MEMS)mirror 1600, according to some embodiments. In some embodiments, a lightengine may include four scanning mirrors, with two used for fovealprojection, one mirror 1600 for each pair of foveal projectors, and twoused for peripheral projection, one mirror 1600 for each pair ofperipheral projectors. In some embodiments, each scanning mirror 1600operates at a resonant frequency of 22 kHz. However, scanning mirrors1600 with other resonant frequencies may be used, for example mirrors1600 that operate at a resonant frequency of 30 kHz or higher for fastaxis scans. Note that using mirrors 1600 that operate at a resonantfrequency higher than 22 kHz (e.g., 30 kHz) may allow a reduction fromfour laser diodes per color per foveal projector down to two laserdiodes per color per foveal projector. Thus, the laser array projectoras illustrated in FIGS. 15 and 16 may be reduced from 20 laser diodes ofeach color down to 12 laser diodes of each color (2 for each fovealprojector and 1 for each peripheral projector), and thus from 60 laserdiodes total down to 36 laser diodes total, with corresponding changesin the configuration of the distribution waveguide 1500.

Light exiting the scanning mirrors enters corresponding foveal andperipheral optical waveguides. In some embodiments, there may be twowaveguides for foveal projection, with a first waveguide for two of thefoveal projectors, and a second waveguide for the other two fovealprojectors. In some embodiments, there may be four waveguides forperipheral projection, with a peripheral waveguide for each of theperipheral projectors.

FIG. 17 illustrates layered foveal waveguides with pupil expansion,according to some embodiments. There may be two 2D scanning MEMS mirrors1600 for foveal projection, each scanning light from the distributionwaveguide into pupil expansion gratings of a respective foveal opticalwaveguide 1700 for a pair of foveal projection points (fovealprojectors). For example, scanning mirror 1600A scans for fovealprojectors 1 and 4 on foveal waveguide 1700A, and scanning mirror 1600Bscans for foveal projectors 2 and 3 on foveal waveguide 1700B. Aspreviously noted, each foveal projector may be illuminated by 12 laserdiodes of the light engine; the 12 laser diodes include 4 laser diodesof each color/wavelength (red, green, and blue (RGB)). Using 4 laserdiodes for each color to illuminate a foveal projector may reduce therequired scan angle of the scanning MEMS mirrors 1600, and hence reducethe mirror speed required when scanning pixels. Higher mirror speeds maytend to smear pixels, which reduces resolution.

In some embodiments, the gratings on the foveal waveguides 1700 havevertical grating vectors (in the +Y direction). In some embodiments, thegratings have a 700 nm grating spacing and allow diffraction in the +1,0 and −1 diffraction orders. In some embodiments, there is no pupilexpansion grating on one side of the exit gratings at the fovealprojectors, and so the projected beams from the MEMS mirrors 1600 arenot at the required elevation angles (azimuth scan angles are correct).Hence, the exit aperture may also require a weak grating to correct theelevation angles.

In some embodiments, for projectors 1 and 4, all 0 order beams areangled downwards; for projectors 2 and 3, all 0 order beams are angledupward. This allows the output aperture to be filled. However, this maydepend on the grating efficiencies for the different orders, whichideally needs to vary across the pupil expansion grating.

In some embodiments, the pupil expansion gratings operate by diffractinglight into the different orders (−1, 0, +1) as the light propagatestowards the output grating; however, to exit, the light must be at order0, meaning light must have diffracted in the +1 direction the samenumber of times it diffracted in the −1 direction. Light that does notmeet this condition will not emerge from the exit grating.

In some embodiments, each exit grating has a spacing of 5000 nm.Projectors 1 and 4 diffract into the +1 order. Projectors 2 and 3diffract into the −1 order.

FIG. 18 illustrates beam angles in cosine space for a foveal waveguideas illustrated in FIG. 17, according to some embodiments. The diagramsshow the beam angles in cosine space as they propagate through thefoveal waveguide. Since there is an exit grating that is not matchedelsewhere, the three colors diffract differently. Hence to compensate,the input scan onto the MEMS mirror may be at different angles for thethree colors. The output scan should have RBG correctly aligned. Sincethe pupil expansion diffractions into the +1 and −1 orders do no overlapthe 0 order, there is no possibility of light from these orders enteringthe eye in the projected FOV.

FIG. 19 illustrates peripheral waveguides, according to someembodiments. In some embodiments, there may be four (e.g., 100 um thick)peripheral waveguides 1910 laminated together. In some embodiments,there may be two 2D scanning MEMS mirrors 1920 for peripheralprojection, each scanning light from the distribution waveguide intopupil expansion gratings of two of the waveguides 1910. In someembodiments, the scanned rays from each mirror 1920 are spatiallyseparated to ensure they can be directed into the correct waveguide1910, for example using appropriate coatings. In some embodiments, thereis only one diffraction grating per waveguide (a pupil expansiongrating). In some embodiments, there may be coatings between the layeredwaveguides 1910 where needed to ensure that light is captured. In someembodiments, there are apertures in the layered waveguides 1910 at theexit to let light out. In some embodiments, the gratings for the fourperipheral projectors 1900 are identical. Hence, depending on themanufacturing process, it may be possible to record all the gratings(holograms) at the same time. In some embodiments, all of the gratingshave a grating vector in the +Y direction, with a grating spacing of 700nm. Diffraction is allowed into the +1, 0 and −1 orders. In someembodiments, all light enters and exits at the same angle. Light canonly escape if diffracted into the +1 order the same number of times asthe −1 order.

FIG. 20 illustrates beam angles in cosine space for peripheralwaveguides, according to some embodiments. In some embodiments, thepupil expansion grating spacing and scan range may be optimized toensure that all scan angles can diffract into the +1 and −1 orders, andthat the +1 and −1 orders do not overlap with the 0 order.

FIG. 21 further illustrates peripheral waveguides, according to someembodiments. In some embodiments, there may be four peripheralwaveguides 2100 laminated together. Each peripheral waveguide may havean HOE layer recorded with the pupil expansion grating. In someembodiments, there may be coatings between the layers to prevent lightfrom traveling between the waveguides 2100 except at the input andoutput apertures. In some embodiments, the layered waveguides 2100 mayinclude a light absorbing feature that separates light from eachwaveguide at the output apertures.

Holographic Combiner Details

FIGS. 22A through 22C are used to illustrate aspects of using aholographic combiner rather than a reflective mirror as shown in FIG. 4which will take light from any angle and deflect it in a certaindirection. In some embodiments, to simplify the AR system, rather thanmaneuvering the light around with a secondary scanning mirror as shownin FIG. 4, the AR system projects from a given projection point ontomultiple points in the eyebox. In that way, the mechanics of the systemcan be simplified when compared to the system of FIG. 4, while alsodelivering a large eyebox that is tolerant to different people's facegeometry and different eye positions.

FIGS. 22A through 22C are graphs illustrating angular selectivity for apoint-to-point holographic combiner, according to some embodiments. Insome embodiments, peripheral projector angular selectivity may be >1.6deg and <2.2 deg. However, this may be altered depending on constraints.In some embodiments, foveal projector (with larger diameter beams)angular selectivity may be >7.1 deg and <8.6 deg. The constraints aredifferent because the beam sizes are different. In some embodiments,angular selectivity for a holographic element may be altered byadjusting its film thickness so that it behaves more or less like avolume hologram.

FIG. 22A shows light from three different projection points projectingonto the holographic combiner; holograms on the combiner redirect thelight to three different points in the eyebox. Because of eye movement,rotation, pupil spacing, etc., the pupil of the subject's eye may be atdifferent points in the eyebox. The AR system as described herein may beable to identify the location of the subject's pupil in the eyebox(e.g., using gaze tracking technology), and selectively project lightonto those different points in the eyebox.

In some embodiments, to accomplish this, a series of holograms arerecorded onto the holographic combiner that are configured to redirectlight from particular projection points to particular points in theeyebox. Thus, light can be directed to particular eyebox points byemitting light from different projection points.

FIG. 22B shows graphically some implications of different projectionpoints. For a given point on the holographic combiner, respectiveholograms need to be selective enough so that if the projection angle ischanged, the light will be diffracted from one hologram, or another, orthe next. Point-to-point holograms have selectivity, as illustrated inFIG. 22C which specifically addresses a relationship between thicknessand selectivity of holograms. A thin hologram may be very similar to aregular surface diffraction grating, which will diffract light coming inat any angle, and diffract it into another angle. However, thickerholograms become more like a volume hologram, and can be programmed suchthat only light from a certain angle gets diffracted; light outside thatangular range passes through. Generally, the thicker the hologram, themore selective for angle. Thus, the thickness of a hologram is highlyinfluential in terms of the angular selectivity.

Based on the above analysis, and referring again to FIG. 22A, the actualdifferences between angles between the rays from the three projectionpoints is roughly 2-2.5 degrees as the rays move from those projectionpoints, as illustrated in FIG. 22B. The holographic combiner's hologramsneed to be sensitive enough so that they distinguish between thedifferent projection points and project light in the right direction.Thus, in some embodiments, based on the analysis of FIG. 22C, somethingin the order of a 70 micron thick hologram may be used so that theangular selectivity that is needed is achieved by the holographiccombiner.

FIG. 23 illustrates foveal projections for a holographic combiner,according to some embodiments. Four foveal eye box points are shownilluminated at the same time from a single foveal projection point.However, in some embodiments, two or three rows of foveal eye box points(i.e., 8 or 12 eye box points) may be simultaneously illuminated from asingle foveal projection point.

FIG. 24 illustrates peripheral projections for a holographic combiner,according to some embodiments. Six peripheral eye box points are shownilluminated at the same time from a single peripheral projection point.However, in some embodiments, three or four rows of peripheral eye boxpoints (i.e. 18 or 24 eye box points) may be simultaneously illuminatedfrom a single peripheral projection point.

FIGS. 25 through 28 illustrate beam focusing in an AR system, accordingto some embodiments. As previously mentioned, because the holographiccombiner has optical power, the beams of light that are projected by thelight engine may need to be focused. A projector projects onto multipleeyebox points at the same time. However, the beam focus required for thedifferent eyebox points is different.

FIG. 25 illustrates a best fit focus curve and a focusing element forperipheral projections in an AR system, according to some embodiments.FIG. 25 shows that there are optimal or ideal focus curves for eachperipheral eye box point 2422. However, in some embodiments, a beamfocusing element 2430 (e.g., implemented as an optical lens oralternatively as holograms for each color) may be used to focus beamsfrom peripheral projection points at a focus curve 2440 that is a ‘bestfit’ of the family of ideal focus curves for the different peripheraleye box points. In some embodiments, the focus curve 2440 may be “bestfit” to provide optimal resolution and less error at the middle of theFOV. More error and lower resolution can be tolerated at the edge of theFOV than at the middle of the FOV. Thus, the best fit focus curve 2440forms the ideal image plane for a notional lens. Beam focusing element2430 focuses the light at best fit focus curve 2440 as the light isscanned across an angle.

In some embodiments, the beam focusing element 2430 may be implementedas a planar holographic optical element. Beam focusing element 2430 maythus be effectively a single component which can be recorded atdifferent spatial points with different holograms for the differentprojectors. In some embodiments, there are eight projectors, four fovealand four peripheral projectors; over the respective projectors, beamfocusing element 2430 will be recorded with holograms that will act aslenses. Thus, beam focusing element 2430 may act as a lens which changesfocus position as light is scanned across the element.

Note that there are errors between the best fit curve 2440 and theoptical focus curves. In some embodiments, to address these errors, thebeam diameter may be changed as light is scanned across the element. Asthe beam diameter is changed, this effectively changes the F-number ofthe system as light enters the eye. As the beam diameter is reduced, theF-number is increased, which increases the depth of focus of the system.Increasing the depth of focus compensates for the errors in focusingwhich may result from the errors between the best fit curve 2440 and theoptical focus curves.

FIG. 26 is a graph of peripheral projector resolution vs. pupil angle inan AR system, according to some embodiments. The graph represents ananalysis using four eye box points across the eye box from a givenprojector point. However, other numbers of eye box points (e.g., six)may be used, and a similar analysis may be done using that other numberof eye box points.

As can be seen in FIG. 26, there are errors between the ideal beam focusfor any given eye box position and the ‘best fit’ curve, whichrepresents the actual beam focus. In some embodiments, the errors can becompensated by adjusting the beam diameter for different field angles,so as to increase the depth of focus by increasing the F-number. In someembodiments, this may be done discretely with the beam diameter being0.5 mm near the center of the scan reducing to 0.35 mm and then 0.22 mmat the extreme ends of the scan. In this way the resolution is optimizedand gracefully reduces at higher angles. As can be seen in FIG. 26, itis possible to realize the peripheral projector with a resolution thatremains competitive with what the eye can actually resolve at higherfield angles.

In FIG. 26, the black line represents eye resolution and how thatchanges over the FOV. The point in the middle represents fovealresolution; resolution drops off at higher field angles. For peripheralprojection, it may not be necessary to project light at high resolutionsat the small angles; instead, a goal is to project light over a muchbigger FOV. FIG. 26 illustrates what happens, when the beam diameter isreduced at higher field angles to compensate for the focusing errors asdescribed above. There is a reduction in resolution at higher fieldangles (all lines are dropping), but for the most part stay above eyeresolution. Thus, errors in focal position can be compensated byaltering the beam diameter as light is scanned. In some embodiments, toalter the beam diameter, holograms that reject light at different anglesmay be used to provide an effective aperture as the light is scanned.

FIG. 27 illustrates a best fit focus curve and a focusing element forfoveal projections in an AR system, according to some embodiments. FIG.27 shows that there are optimal or ideal focus curves for each fovealeye box point 2622. However, in some embodiments, a beam focusingelement 2630 (e.g., implemented as an optical lens or alternatively asholograms for each color) may be used to focus beams from fovealprojection points at a focus curve 2640 that is a ‘best fit’ of thefamily of ideal focus curves for the different foveal eye box points. Insome embodiments, the focus curve 2640 may be “best fit” to provideoptimal resolution and less error at the middle of the FOV. The best fitfocus curve 2640 forms the ideal image plane for a notional lens. Beamfocusing element 2630 focuses the light at best fit focus curve 2640 asthe light is scanned across an angle. Note that the best fit curve isgenerally closer to the optical curves and errors between the best fitcurve and the family of optical curves for foveal each eye box pointsare smaller than those for peripheral eye box points because the fieldof each eye box is smaller and scan angles are smaller. However, errorsare more significant for foveal projection as higher resolution isneeded.

FIG. 28 illustrates projector scan angle for foveal projections,according to some embodiments. FIG. 29A is a graph of foveal projectorresolution vs. pupil angle in an AR system, according to someembodiments. FIG. 29B is a graph of beam diameter for foveal projectionsin an AR system, according to some embodiments. As with the peripheralprojector, it may be necessary to reduce the beam diameter at the edgesof the scan for each foveal eye box point, for example from 2.3 mm atthe center of the scan to a minimum 0.73 mm for a particular range ofscan angles. The optimal resolution may require bands of different beamdiameters across the projector scan range. In some embodiments, coatingson the foveal projectors may be used to achieve this adjustment of beamdiameter with field angle. In some embodiments, the coatings may be partof a further layer laminated to the waveguide structure of the lightengine.

As shown in FIG. 29B, in some embodiments, when scanning through theangles in foveal projection, there will be a “sawtooth” change in beamdiameter. In peripheral projection, the beam diameter can be tailed offat higher angles; for foveal projection, instead, scanning goes througha series of steps, with a large (e.g., 7 mm) beam at the middle of eachscan, but tailing off at bigger angles. As shown in FIG. 29A, fovealprojection is thus constantly above actual eye resolution over theangles of interest, up to 10 degrees, and thus the desired resolutioncan be achieved.

The methods described herein may be implemented in software, hardware,or a combination thereof, in different embodiments. In addition, theorder of the blocks of the methods may be changed, and various elementsmay be added, reordered, combined, omitted, modified, etc. Variousmodifications and changes may be made as would be obvious to a personskilled in the art having the benefit of this disclosure. The variousembodiments described herein are meant to be illustrative and notlimiting. Many variations, modifications, additions, and improvementsare possible. Accordingly, plural instances may be provided forcomponents described herein as a single instance. Boundaries betweenvarious components, operations and data stores are somewhat arbitrary,and particular operations are illustrated in the context of specificillustrative configurations. Other allocations of functionality areenvisioned and may fall within the scope of claims that follow. Finally,structures and functionality presented as discrete components in theexample configurations may be implemented as a combined structure orcomponent. These and other variations, modifications, additions, andimprovements may fall within the scope of embodiments as defined in theclaims that follow.

What is claimed is:
 1. A system, comprising: a controller; a lightengine including a distribution waveguide configured to receive lightbeams from a plurality of light sources at a plurality of entrancepoints and direct the light beams to pass through the distributionwaveguide to a plurality of exit points to project the light beams froma plurality of projection points under control of the controller; and areflective holographic combiner comprising a plurality of point-to-pointholograms configured to redirect light received from the plurality ofprojection points to a plurality of eye box points; wherein eachprojection point projects light beams simultaneously to two or more ofthe plurality of holograms, wherein the two or more holograms redirectthe light beams to illuminate two or more respective eye box points;wherein the holograms are configured such that neighboring eye boxpoints are illuminated by different ones of the holograms, and whereinthe entrance points and exit points of the distribution waveguide areimplemented as holograms using a holographic film or as surface reliefgratings.
 2. The system as recited in claim 1, wherein the plurality ofeye box points includes foveal and peripheral eye box points, andwherein the plurality of projection points includes: two or more fovealprojectors configured to project wide diameter light beams over a smallfield of view, wherein foveal light beams are redirected to illuminatefoveal eye box points; and two or more peripheral projectors configuredto project narrow diameter light beams over a wide field of view,wherein peripheral light beams are redirected to illuminate peripheraleye box points.
 3. The system as recited in claim 2, wherein thediameters of the foveal light beams are 4 mm or greater when exiting thefoveal projectors.
 4. The system as recited in claim 2, wherein thediameters of the foveal light beams are 2.3 mm or less at the foveal eyebox points, and wherein the diameter of the peripheral light beams is0.5 mm or less at the peripheral eye box points.
 5. The system asrecited in claim 2, wherein field of view of the foveal light beams is20° horizontal×20° vertical, and wherein field of view of the peripherallight beams is 120° horizontal×74° vertical.
 6. The system as recited inclaim 2, wherein the light engine includes a plurality of light sources,and wherein the controller is configured to selectively activate andmodulate particular ones of the plurality of light sources to projectlight from different ones of the foveal and peripheral projectors. 7.The system as recited in claim 6, wherein the light sources are arraysof edge-emitting laser diodes in a laser array projector component ofthe light engine.
 8. The system as recited in claim 6, wherein the lightsources include red, green, and blue light sources.
 9. The system asrecited in claim 1, wherein the light engine includes a plurality ofscanning mirrors configured to receive light at appropriate angles andpositions from the plurality of exit points and scan the light to aplurality of layered waveguides that include film layers recorded withholographic or diffractive gratings, wherein the foveal and peripheralprojectors are implemented by respective foveal and peripheralwaveguides of the plurality of layered waveguides.
 10. The system asrecited in claim 9, wherein the foveal and peripheral waveguides includepupil expansion gratings configured to expand the light received fromthe scanning mirrors and direct the light to the foveal and peripheralprojectors, wherein the foveal and peripheral waveguides are configuredto emit the expanded light from the foveal projectors.
 11. The system asrecited in claim 2, wherein the light engine further comprises focusingelements for each projector, wherein the focusing elements areconfigured to focus the light beams emitted by the projectors at focussurfaces in front of the holographic combiner so that the light beamsare substantially collimated when reflected to the eye box points by theholographic combiner.
 12. The system as recited in claim 11, whereinideal focus surfaces are different for different eye box points, andwherein the light engine is configured to reduce a light beam diameterat different projection angles to compensate for errors between thefocus surfaces and the ideal focus surfaces, wherein reducing thediameter of a light beam increases a F-number thus increasing a depth offocus of the light beam.
 13. The system as recited in claim 11, whereinthe focusing elements include holographic lenses.
 14. The system asrecited in claim 1, wherein the system further includes a gaze trackingcomponent configured to track a position of a subject's eye, wherein thecontroller is configured to selectively activate and modulate particularones of a plurality of light sources to selectively illuminateparticular ones of the plurality of eye box points.
 15. The system asrecited in claim 1, wherein the light engine includes a plurality ofscanning mirrors, wherein the scanning mirrors include 2D scanningmicroelectromechanical systems (MEMS) mirrors.
 16. A method, comprising:emitting, by a laser array projector comprising a plurality of lightsources, light beams to a plurality of entrance holograms of adistribution waveguide under control of a controller; guiding, by thedistribution waveguide, the light beams to pass through the distributionwaveguide from the plurality of entrance holograms to respective ones ofa plurality of exit holograms of the distribution waveguide; emitting,at the exit holograms, the light beams to a plurality of scanningmirrors; scanning, by the scanning mirrors, the light beams to layeredwaveguides with pupil expansion; expanding, by pupil expansion gratingsof the layered waveguides, the light beams; projecting, by the layeredwaveguides, the expanded light beams from respective ones of a pluralityof projection points; focusing, by focusing elements, the projectedlight beams in front of a holographic combiner; and redirecting, by aplurality of point-to-point holograms of the holographic combiner, thelight beams to respective ones of a plurality of eye box points.
 17. Themethod as recited in claim 16, wherein the plurality of eye box pointsincludes foveal and peripheral eye box points, and wherein the pluralityof projection points includes: two or more foveal projectors configuredto project wide diameter light beams over a small field of view, whereinfoveal light beams are redirected by respective point-to-point hologramsto illuminate foveal eye box points; and two or more peripheralprojectors configured to project narrow diameter light beams over a widefield of view, wherein peripheral light beams are redirected byrespective point-to-point holograms to illuminate peripheral eye boxpoints; wherein the diameter of the foveal light beams is 4 mm orgreater when exiting the foveal projectors, wherein the diameters of thefoveal light beams are 2.3 mm or less at the foveal eye box points, andwherein the diameters of the peripheral light beams are 0.5 mm or lessat the peripheral eye box points.
 18. The method as recited in claim 17,further comprising selectively activating and modulating, by thecontroller, particular ones of the plurality of light sources to projectlight from different ones of the foveal and peripheral projectors. 19.The method as recited in claim 16, wherein the plurality of eye boxpoints includes foveal and peripheral eye box points, wherein theplurality of projection points includes four foveal projectorsconfigured to project wide diameter light beams that are redirected byrespective point-to-point holograms to illuminate foveal eye box pointsand four peripheral projectors configured to project narrow diameterlight beams that are redirected by respective point-to-point hologramsto illuminate peripheral eye box points.
 20. The method as recited inclaim 16, wherein the plurality of light sources includes red, green,and blue edge emitting lasers, wherein the plurality of light sourcesincludes a plurality of peripheral projectors and a plurality of fovealprojectors, wherein each peripheral projector is illuminated by one red,one green, and one blue laser, and wherein each foveal projector isilluminated by four red, four green, and four blue lasers.